Method and system for assessing damage to infrastructure

ABSTRACT

A method and system may survey a property using aerial images captured from an unmanned aerial vehicle (UAV), a manned aerial vehicle (MAV) or from a satellite device. The method may include identifying a commercial property for a UAV to perform surveillance, and directing the UAV to hover over the commercial property and capture aerial images at predetermined time intervals. Furthermore, the method may include receiving the aerial images of the commercial property captured at the predetermined time intervals, detecting a surveillance event at the commercial property, generating a surveillance alert, and transmitting the surveillance alert to an electronic device associated with an owner of the commercial property.

RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S.application Ser. No. 15/966,086 filed on Apr. 30, 2018, entitled “Methodand System For Assessing Damage To Infrastructure,” which is acontinuation of and claims priority to U.S. application Ser. No.15/718,323 filed on Sep. 28, 2017, entitled “Method and System ForAssessing Damage To Infrastructure,” which is a continuation of andclaims priority to U.S. application Ser. No. 15/165,457 filed on May 26,2016, entitled “Method and System For Assessing Damage ToInfrastructure,” which is a continuation of and claims priority to U.S.application Ser. No. 14/808,502 filed on Jul. 24, 2015, entitled “Methodand System For Assessing Damage To Infrastructure,” which is acontinuation of and claims priority to U.S. application Ser. No.14/510,784, filed on Oct. 9, 2014, entitled “Method and System ForAssessing Damage To Infrastructure,” the entire disclosures of each ofwhich are hereby expressly incorporated by reference.

TECHNICAL FIELD

The present disclosure generally relates to a method and system forassessing damage to infrastructure such as roads, highways, bridges,etc.

BACKGROUND

Infrastructure such as roads, highways, bridges, etc., often becomesdamaged over time due to wear and tear, natural disasters, extremeweather conditions, etc. To perform regular maintenance on theinfrastructure, personnel may need to travel to the site to determinehow to repair or replace the damaged infrastructure.

Often, the investigations can be time-consuming, difficult and evendangerous for the on-site personnel. For example, in order toinvestigate the damage to a highway, an inspector may have to travel tothe highway and inspect the condition of the road. While a lane orsection of the highway may be blocked off to prevent vehicles frompassing by during the inspection, some vehicles may accidentally crossthese barriers and crash into the inspectors performing theinvestigation, resulting in injuries or even death.

Even if the inspectors perform the investigation without gettinginjured, performing the full investigation may still be time-consuming.In addition to the time required to drive to and from the site and toperform the inspection itself, significant paperwork and calculationsmay be involved in calculating the cost of repairing the item ofinfrastructure. For example, if an inspector takes photos on the site toassess the amount of damage to the highway, the inspector may have tocome back to her office, research the cost of the damaged infrastructureitem and research repair costs. All of these steps are time consumingand both delay repairs and prevent the inspector from assessing damageto other items of infrastructure.

SUMMARY

To assess the extent or severity of the damage to infrastructure, anautomated infrastructure evaluation system may identify an item ofinfrastructure for assessing damage. For example, a bridge, a road, ahighway, a tunnel, a sewer treatment plant, a water treatment plant, areservoir, an aqueduct, an electric power grid, a communications tower,a sidewalk, a paved walkway, a rail line, a waterway (e.g., locks anddams), a port facility, a public transportation system, etc., may beidentified. The infrastructure evaluation system may also determineboundaries for assessing the damage to the item of infrastructure. Forexample, if a stretch of Highway 80 from exit 220 to exit 225 is to beevaluated, the system may identify a set of boundaries (e.g., globalpositioning system (GPS) coordinates) which encapsulates the areabetween exit 220 and exit 225 on Highway 80.

Using the identified boundaries, the system may perform an automaticinspection of the infrastructure item. The automatic inspection may beperformed by an unmanned aerial vehicle (UAV), or by a swarm of UAVsworking together, which may be controlled by an inspector or by thesystem and flown within the specified boundaries to capture aerialimages of the item. Alternatively, the automatic inspection may beperformed by a satellite which also captures aerial images of theinfrastructure item for the specified boundaries. Moreover, theinspection may also be performed by a manned aerial vehicle (MAV) whichcaptures aerial images of the infrastructure item. Each captured aerialimage may be associated with a location, for example a GPS location, andthe GPS location may be used to aggregate the aerial images to form a3-dimensional (3D) image.

The aerial images may also be analyzed to determine the condition of theinfrastructure item as well as the extent or severity of the damage tothe infrastructure item. Moreover, the system may assign an indicator(e.g., a color from a set of colors), which indicates the extent of thedamage to the infrastructure item or to a specific portion of theinfrastructure item. The system may also display the assigned indicatoralong with the infrastructure item on a computing device for a user,such as the inspector, to observe. In this manner, the damage assessmentfor infrastructure can be performed automatically, without requiring aninspector to spend her time and risk injury investigating the damage.Moreover, the system provides indicators to allow an inspector toquickly and easily view areas where the damage is most severe in orderto determine costs and the necessary repairs for the severely damagedareas.

In an embodiment, a method for surveying a property using an unmannedaerial vehicle (UAV) is provided. The method includes identifying acommercial property for a UAV to perform surveillance, and directing theUAV to hover over the commercial property and capture aerial images atpredetermined time intervals. Furthermore, the method includes receivingthe aerial images of the commercial property captured at thepredetermined time intervals, detecting a surveillance event at thecommercial property, generating a surveillance alert, and transmittingthe surveillance alert to an electronic device associated with an ownerof the commercial property.

In another embodiment, a system for surveying a property using anunmanned aerial vehicle (UAV) is provided. The system includes one ormore processors, a communication network, and a non-transitorycomputer-readable memory coupled to the one or more processors, and thecommunication network, and storing instructions thereon. When executedby the one or more processors, the instructions cause the system toidentify a commercial property for a UAV to perform surveillance, anddirect the UAV to hover over the commercial property and capture aerialimages at predetermined time intervals. The instructions further causethe system to receive the aerial images of the commercial propertycaptured at the predetermined time intervals, detect a surveillanceevent at the commercial property, generate a surveillance alert, andtransmit the surveillance alert to an electronic device associated withan owner of the commercial property.

In yet another embodiment, a non-transitory computer-readable memorycoupled to one or more processors and storing instructions thereon isprovided. When executed by the one or more processors, the instructionscause the one or more processors to identify a commercial property for aUAV to perform surveillance, and direct the UAV to hover over thecommercial property and capture aerial images at predetermined timeintervals. The instructions further cause the one or more processors toreceive the aerial images of the commercial property captured at thepredetermined time intervals, detect a surveillance event at thecommercial property, generate a surveillance alert, and transmit thesurveillance alert to an electronic device associated with an owner ofthe commercial property.

BRIEF DESCRIPTION OF THE DRAWINGS

The figures described below depict various aspects of the system andmethods disclosed therein. It should be understood that each figuredepicts an embodiment of a particular aspect of the disclosed system andmethods, and that each of the figures is intended to accord with apossible embodiment thereof. Further, wherever possible, the followingdescription refers to the reference numerals included in the followingfigures, in which features depicted in multiple figures are designatedwith consistent reference numerals.

FIG. 1A illustrates a block diagram of an example system in whichtechniques for performing an automatic damage assessment of an item ofinfrastructure are implemented;

FIG. 1B illustrates a block diagram of an exemplary mobile device;

FIG. 2 illustrates a block diagram detailing an exemplary embodiment ofan image receiving module;

FIG. 3 depicts a block diagram detailing an exemplary embodiment of aninfrastructure condition assessment module;

FIG. 4 depicts a block diagram detailing an exemplary embodiment of aninfrastructure damage severity determination module;

FIG. 5 depicts an exemplary display of an infrastructure item includingdamage severity level indicators; and

FIG. 6 illustrates a flow diagram representing an exemplary method forperforming automatic damage assessment of an item of infrastructure inaccordance with the presently described embodiments.

DETAILED DESCRIPTION

Although the following text sets forth a detailed description ofnumerous different embodiments, it should be understood that the legalscope of the description is defined by the words of the claims set forthat the end of this patent and equivalents. The detailed description isto be construed as exemplary only and does not describe every possibleembodiment since describing every possible embodiment would beimpractical. Numerous alternative embodiments could be implemented,using either current technology or technology developed after the filingdate of this patent, which would still fall within the scope of theclaims.

It should also be understood that, unless a term is expressly defined inthis patent using the sentence “As used herein, the term ‘______’ ishereby defined to mean . . . ” or a similar sentence, there is no intentto limit the meaning of that term, either expressly or by implication,beyond its plain or ordinary meaning, and such term should not beinterpreted to be limited in scope based on any statement made in anysection of this patent (other than the language of the claims). To theextent that any term recited in the claims at the end of this patent isreferred to in this patent in a manner consistent with a single meaning,that is done for sake of clarity only so as to not confuse the reader,and it is not intended that such claim term be limited, by implicationor otherwise, to that single meaning. Finally, unless a claim element isdefined by reciting the word “means” and a function without the recitalof any structure, it is not intended that the scope of any claim elementbe interpreted based on the application of 35 U.S.C. § 112, sixthparagraph.

Accordingly, the term “aerial image” as used herein, may be used torefer to any image data within the electromagnetic spectrum (i.e.including the visible light spectrum as well as the invisible lightspectrum), which is captured from an elevated position. Aerial imagesmay include visible light imaging, radar imaging, near infrared imaging,thermal infrared imaging, hyperspectral imaging, multispectral imaging,full spectral imaging, etc. For example, an image captured by asatellite, a manned aerial vehicle (MAV) or an unmanned aerial vehicle(UAV) may be referred to herein as an “aerial image.” An aerial imagemay be made up of data points, for example pixel data points, where eachdata point may correspond to a specific global positioning system (GPS)location. An aerial image may also include video captured from anelevated position.

Also, the term “infrastructure item” or “item of infrastructure” as usedherein, generally refers to a physical component that providescommodities and/or services essential to enable, sustain, or enhancesocietal living conditions. An infrastructure item may include ahighway, a road, a bridge, a tunnel, a sewer treatment plant, a watertreatment plant, a reservoir, an aqueduct, an electric power grid, acommunications tower, a sidewalk, a paved walkway, a rail line, awaterway (e.g., locks and dams), a port facility, a publictransportation system, etc., or any portion thereof. In someembodiments, items of infrastructure may not include buildings.

Generally speaking, to perform the automatic infrastructure evaluationprocess, an aerial image capturing device which may be a satellite, MAV,or one or several UAV(s) is/are directed to capture images within aspecified set of boundaries which encapsulates an identified item ofinfrastructure (e.g., a highway segment between exit 10 and exit 20).The aerial image capturing device may be directed by a client devicehaving user controls for determining the location and the amount ofphotographs or video captured. The captured aerial images may then beprovided to the client device or to a server computer. The aerial imagesmay be aggregated, for example using photogrammetry, stereoscopy, orLIDAR, to create a 3D image of the identified item.

The 2D or 3D image may be displayed on the client device and may becreated at a predefined level of detail (e.g., accurate to within tenpercent) and/or may be adjustable (e.g., a user of the system may beable to “zoom in” or “zoom out” of the image). Moreover, the 2D or 3Dimage may be made up of data points, for example pixel data points,where each data point may correspond to a specific global positioningsystem (GPS) location. Each of these data points may then be comparedwith other images of the same type of infrastructure for determining thecondition of the infrastructure item depicted by the data points. Forexample, an image of a bridge may be compared to an image of a perfectlyintact bridge. The condition may be used to determine the severity ofthe damage to the item of infrastructure or to a portion of the item ofinfrastructure. This damage severity level may be provided to aninspector, for example, on the user interface of the client device, fordetermining a cost of repair and/or replacement. Alternatively, theclient device or the server computer may automatically determine thecost of repair and/or replacement based on the damage severity leveland/or the aerial images.

FIG. 1A illustrates various aspects of an exemplary environmentimplementing an automated infrastructure evaluation system 10 (alsoreferred to herein as “the system”). The system 10 may include a clientdevice 12 with remote control capabilities coupled to one or severalunmanned aerial vehicles (UAVs) 40, an MAV, a satellite device 18, and aserver 14 via a communication network 16. The client device 12 may be,for example, a laptop computer, a tablet computer, a smartphone, awearable device, etc. In the embodiment illustrated in FIG. 1A, theclient device 12 may include a central processing unit (CPU) 20, agraphics processing unit (GPU) 22, a computer-readable memory 24, and afirst user interface 30 for controlling the UAV(s) 40 or the satellitedevice 18. The first user interface 30 may include a touch interface 32,voice interface 34, etc. In various implementations, the touch interface32 can include a touchpad over which the user moves his fingers whilelooking at a separately provided screen, a touchscreen where the userplaces his fingers directly over the image being manipulated or over adisplayed control being activated (e.g. a displayed keyboard), etc. Inother implementations, the voice interface 34 may include any devicethat includes a microphone, such as a Bluetooth ear piece, a smartphone,etc. The client device 12 may further include a second user interface 70which may be used for viewing aerial images captured by the UAV(s) 40,MAV, or the satellite device 18. In some embodiments, the first userinterface 30 and the second user interface 70 may be implemented on oneuser interface which includes user controls for directing the UAV(s) 40or the satellite device 18 and displays the aerial images after theaerial images have been captured.

The memory 24 is a computer-readable non-transitory storage device thatmay include both persistent (e.g., a hard disk) and non-persistent(e.g., RAM) memory components, stores instructions executable on the CPU20 and/or the GPU 22 that make up an infrastructure evaluation module(IEM) 72, a remote control module 36 and location data 26 and sensordata 28 on which the remote control module 36 operates. The remotecontrol module 36 includes an incremental movement module 38 that allowsa user to easily control the UAV(s) 40 via step-like, incrementalmovements in which one incremental movement is in response to one singleuser command.

The remote control module 36 and the infrastructure evaluation module 72according to various implementations operate as separately executablesoftware applications, plugins that extend the functionality of anothersoftware application such as a web browser, application programminginterfaces (API) invokable by a software application, etc. Theinstructions that make up the remote control module 36 and theinfrastructure evaluation module 72 may be compiled and executable onthe CPU 20 and/or the GPU 22 directly, or not compiled and interpretedby the CPU 20 at runtime. However, FIG. 1A merely illustrates acondensed version of the client device 12, and a more detailed versionof the client device 12 is described below with reference to FIG. 1B.

Referring still to FIG. 1A, each UAV 40 includes a controller 42 thatcommunicates with one or more proximity sensors 44, one or morestabilization sensors 45, a Global Positioning System (GPS) unit 46,image sensors 47, and a communications unit 48. The image sensors 47 mayinclude one or more filters for infrared imaging, hyperspectral imaging,multispectral imaging, full spectral imaging, etc., or alternatively,the image sensors 47 may include one or more sensors which receive imagedata outside of the visible light spectrum such as an infrared imagesensor. The controller 42 includes a processor 50 that executesinstructions from a computer-readable memory 52 to implement a controlmodule 54 and a stabilization module 56. The control module 54 mayinvoke the stabilization module 56 to retrieve data from thestabilization sensors 45 (i.e., sensors relating to avionics) toimplement a control function, such as that associated with a controlroutine that performs PID (proportional-integral-derivative), fuzzylogic, nonlinear, etc. control to maintain the stability of the UAV(s)40. For instance, the stabilization sensors 45 may include one or moreof a directional speed sensor, a rotational speed sensors, a tilt anglesensor, an inertial sensor, an accelerometer sensor, or any othersuitable sensor for assisting in stabilization of an aerial craft. Ofcourse, the stabilization module 56 may implement any suitable techniqueof stabilizing the remote aerial device 40 in a hover or stationarythree dimensional position.

The control module 54 may retrieve data from the proximity sensors 44.These proximity sensors 44 may include any sensor or technique thatassists the control module 44 in determining a distance and a directionto the infrastructure. The one or more proximity sensors 44 may includeoptic flow sensors, ultrasonic sensors, infrared sensors, LIDAR (LightDetection and Ranging), a stereo vision system (SVS) that may utilizethe image sensors 47 (e.g., one or more cameras) to implementstereoscopic imaging techniques to capture aerial images of theinfrastructure item and to create 3D images of the infrastructure item.The control module 54 may also receive instructions from the clientdevice 12 to capture aerial images at specific locations or timeintervals.

The GPS unit 46 may use “Assisted GPS” (A-GPS), satellite GPS, or anyother suitable global positioning protocol or system that locates theposition of the UAV(s) 40. Moreover, the GPS unit 46 may also determinethe position of the aerial images or of data points within the aerialimages captured by the UAV(s) 40, or the GPS may be combined with thedistance and direction sensors 44 to determine the position of theaerial images, and positions of data points within an aerial image. Forexample, A-GPS utilizes terrestrial cell phone towers or wi-fi hotspots(e.g., wireless router points) to more accurately and more quicklydetermine the location of the device while satellite GPS generally aremore useful in more remote regions that lack cell towers or wi-fihotspots. The communication unit 48 may communicate with the server 14or the client device 12 via any suitable wireless communication protocolnetwork, such as a wireless telephony network (e.g., GSM, CDMA, LTE,etc.), a wi-fi network (802.11 standards), a WiMAX network, a Bluetoothnetwork, etc.

As mentioned above, the system 10 may also include a satellite device 18which includes an image sensor 82 for capturing aerial images and a GPSunit 84 for determining the position of each image. For example, thesatellite device 18 may determine GPS coordinates of the boundaries ofan aerial image, and also may determine GPS coordinates of data points,such as pixel data points, of the aerial image. The satellite device 18may also include a processor 86 which executes instructions from acomputer-readable memory 88 to implement an image capturing module 90,which may capture and transmit satellite images at the request of theclient device 12. For example, the client device 12 may requestsatellite images between specified GPS coordinates, and the imagecapturing module 90 may transmit satellite images within the specifiedcoordinates. Moreover, in some embodiments the client device 12 mayspecify the number of satellite images for the image capturing module 90to capture and the zoom level. The client device 12 or the server 14 andthe satellite device 18 may communicate via a communication unit 92 viaany suitable wireless communication protocol network, such as a wirelesstelephony network (e.g., GSM, CDMA, LTE, etc.), a wi-fi network (802.11standards), a WiMAX network, a Bluetooth network, etc.

The server 14 may include infrastructure data (e.g., a list of items ofinfrastructure such as “Highway 80,” “the Golden Gate Bridge,” “the ‘L’Station,” etc.), location data (e.g., locations of the items ofinfrastructure, locations of portions of the items of infrastructure,etc.), previous image data (e.g., aerial images of items ofinfrastructure taken at an earlier date), and financial data (e.g.,infrastructure cost estimates of property and materials similar to thosethat were damaged or destroyed, labor costs for repairing/replacing theinfrastructure, etc.) from an infrastructure database 66, a locationdatabase 68, a previous image database 94, and a financial database 96,respectively. The server 14 then may provide the infrastructure data,the location data, the previous image data, the financial data andappropriate indications of how certain portions of the infrastructuredata and the location data are linked, to the client device 12 as partof the location data 26. The client device 12 may use this location datato determine a geographic location that the UAV(s) 40 is/are initiallysent to and may use the previous image data to determine a condition ofan item of infrastructure as compared to its previous condition. Thefinancial data may be used for performing cost estimates for repairinginfrastructure. The infrastructure database 66, the location database68, the previous image database 94 and the financial database 96 may bedisposed within the client device 12 depending on the implementation.The server may also include a processor 60 which executes instructionsfrom a computer-readable memory 62 to implement an infrastructureevaluation module 73, which may be the same as the infrastructureevaluation module 72 of the client device 12. In some embodiments, theinfrastructure evaluation module 72 may be disposed in the client device12, in the server 14 or in a combination of the server 14 and the clientdevice 12.

FIG. 1B illustrates the client device 12 of FIG. 1A in further detail.As illustrated in FIG. 1A, the client device may include a CPU 20, a GPU22, and a memory 24 which may be a hard drive, an optical drive, a solidstate memory device, or any other non-volatile memory device. The clientdevice 12 may further include an input/output (I/O) unit 103 and a datastorage 116, which may include infrastructure data, location data,previous image data, financial data, etc., which may be retrieved fromsoftware instructions which may be stored in the memory 24. Duringexecution, the software instructions may be stored in, and may store andretrieve data from, a volatile or non-volatile memory source, such as arandom access memory (RAM) 106. The client device 12 may include anetwork interface module (NIM) 108 for wired and/or wirelesscommunications. The network interface module 108 may allow the device tocommunicate with one or more other devices such as the server 14, thesatellite device 18, the MAV, or the UAV(s) 40 of FIG. 1A, by using oneor more of any number of communications protocols including, by way ofexample and not limitation, Ethernet, cellular telephony, IEEE 802.11(i.e., “Wi-Fi”), Fibre Channel, etc. The memory 24 may store aninfrastructure evaluation module 72 as described above. Theinfrastructure evaluation module 72 may be a sub-routine of a softwareapplication or may be an independent software routine in and of itself.Alternatively, in some implementations, the infrastructure evaluationmodule 72 may be a hardware module or a firmware module. Theinfrastructure evaluation module 72 may include compiled instructionsdirectly executable by the CPU 20, scripted instructions that areinterpreted at runtime, or both.

The client device may also include a user interface (UI) 118 whichincludes the remote user interface 30 and the image user interface 70 ofFIG. 1A. The remote user interface 30 may include user controls fordirecting the UAV(s) 40 to capture images, for requesting aerial imagesfrom the MAV or for requesting satellite images from the satellitedevice 18 at specific locations. On the other hand, the image userinterface 70 may display aerial images of an infrastructure item and mayalso display damage severity levels for the infrastructure item.

The infrastructure evaluation module (IEM) 72 may contain one or more ofan image receiving module (IRM) 115, an infrastructure conditionassessment module (ICAM) 117, and/or an infrastructure damage severitydetermination module (IDSDM) 119. The IEM 72 may determine the severityof the damage (also referred to herein as a “damage severity level”)associated with an item of infrastructure according to the presentlydescribed techniques. More specifically, the IEM 72 may automaticallydetermine the condition of an item of infrastructure based on stored andreceived aerial images and/or other data describing items ofinfrastructure of the same type (e.g., if the item of infrastructure isa communications tower, the condition is determined based on storedimages of communications towers). The aerial images may be stored in thememory 24 and/or RAM 106. In instances where the IEM 72 executes on aserver device, the damage severity level for an item of infrastructuremay be transmitted to the client device 12. Additionally, the IEM 72 mayperform certain calculations on the server device 14 of FIG. 1A whileother calculations are performed on the client device 12. Moreover, thememory 24, may also include a remote control module 36 and location data26 and sensor data 28 on which the remote control module 36 operates asdescribed above with reference to FIG. 1A.

FIG. 2 is a block diagram detailing an exemplary embodiment of the imagereceiving module 115 according to the present disclosure. The imagereceiving module 115 may include a location designation module 210, andan aggregation module 220. The location designation module 210, and theaggregation module 220 may be separate modules or may be combined andmay interact with each other and/or with other software, hardware,and/or firmware.

The location designation module 210 may identify an item ofinfrastructure for assessing damage. To identify the item ofinfrastructure, the location designation module 210 may connect to athird-party server (not shown). The third-party server can include datafrom news sources (e.g., national news networks, regional news networks,newspapers, magazines, news websites, and others), data from weathersources (e.g., the National Oceanic and Atmospheric Administration;other federal, state, or local governmental weather bureaus; commercialweather services; weather websites; and others), data from governmentalsources (e.g., the Department of the Interior, the Department ofHomeland Security, other federal, state, and local governmental sources,and others), data from social networks (e.g., Facebook®, Twitter®,Google+®, Instagram®, and others), data from public databases, data fromprivate databases (e.g., consultants, data miners, surveyors, andothers), crowd sourced weather data (e.g., connected users or userdevices may report extreme weather conditions to a central server) orother sources. The location designation module 210 may then use thisdata to determine geographic locations where damage to infrastructure islikely to have occurred, and may identify an item(s) of infrastructurewithin the determined geographic locations, for example, using thelocation database 68 of FIG. 1A. Moreover, in some embodiments the userof the client device 12 such as an inspector, may input a name of anitem of infrastructure or a portion thereof such as “Golden Gate Bridge”or “Route 66 in Illinois between exit 40 and exit 102” into the clientdevice 12 which may be provided to the location designation module 210.

In any event, when the infrastructure item is identified, theinfrastructure evaluation module 72 may request and/or receive aerialimages of the identified infrastructure item. For example, theinfrastructure evaluation module 72 may receive the aerial images of theidentified item from the satellite device 18 of FIG. 1A, the MAV, orfrom the UAV(s) 40. The aerial images may be received from the UAV(s) 40by automatically directing the one or several UAV(s) 40 to fly withinthe set of boundaries which encapsulate the identified item. Forexample, the infrastructure evaluation module 72 may look up thelocation of the identified infrastructure item in the location database68 to determine the appropriate set of boundaries. The UAV(s) 40 mayalso be directed to take several photographs or capture video atdifferent locations surrounding the infrastructure item and at severalangles. Alternatively, after the infrastructure item is identified, auser such as an inspector may control the UAV(s) 40 remotely, through aseries of user controls on the remote user interface 30 to cause theUAV(s) to take pictures at different locations surrounding theinfrastructure item and at several angles.

After the aerial images are captured and received for the identifiedinfrastructure item, the infrastructure evaluation module 72 may combinethe aerial images using an aggregation module 220. The aerial images maybe combined to generate a 3D image of the infrastructure item using 3Dimaging techniques such as LIDAR, stereoscopy, or photogrammetry. Theaggregation module 220 may utilize the Cartesian or GPS coordinatesreceived with each aerial image to reconstruct a 3D image of theinfrastructure item using the aerial images captured at differentlocations and angles. In some embodiments, the 3D aerial image may becreated at a predefined level of detail (e.g., accurate to within tenpercent) and/or may be adjustable (e.g., a user or the system may beable to “zoom in” or “zoom out”)

FIG. 3 is a block diagram detailing an exemplary embodiment of theinfrastructure condition assessment module 117. The infrastructurecondition assessment module 117 may include a filtering module 310, acomparison module 320 and a condition determination module 330. In someembodiments, the infrastructure condition assessment module 117 mayobtain a 3D aerial image of an infrastructure item and determine thecondition of the item. In other embodiments, the condition assessmentmodule 117 does not obtain 3D aerial images and instead obtains 2Daerial images. Moreover, in some embodiments, the 3D aerial image may beobtained from the image receiving module 115.

In any event, the filtering module 310 may analyze the received aerialimages and filter out (i.e., remove from consideration for furtheranalysis) one or more irrelevant and/or unexpected data points. Forexample, if the damage assessment is for a road, the aerial images mayinclude data points which display the road as well as data points wheredamage assessment does not need to be performed (e.g., data pointsdepicting nearby side streets, and surrounding terrain such as grass,mountains, bodies of water, the sky, trees, etc.). The filtering module310 may remove the unnecessary data points, so that the unnecessary datapoints are not included and/or considered in the damage assessment forthe road. Various image processing techniques such as edge detection maybe used by the filtering module 310 for determining the unnecessary datapoints of an aerial image(s).

Once the filtering module 310 removes the unnecessary data points fromthe aerial images, the remaining data points may be compared withanother predefined infrastructure item using the comparison module 320.The comparison module 320 may compare the data points of theinfrastructure item with data describing a predefined infrastructureitem of the same type. If the infrastructure item is a bridge, forexample, the comparison module 320 may compare data extracted by thefiltering module 310 with previously stored images of an intact bridge.Based on these comparisons, the comparison module 320 may determinephysical differences between the bridge depicted by the data points andthe intact bridge. For example, the comparison module 320 may determinethat the bridge differs in color (e.g., due to rusting), thickness(e.g., due to cracks or dents in the surface), and/or in shape (e.g.,due to structural damage to the bridge) from the intact bridge.

Moreover, the comparison module 320 may also compare the data pointsdepicting a bridge with a previously stored image(s) of the same bridge,for example, from five years ago when the bridge was known to be in goodcondition. The previously stored image(s) of the same bridge may beobtained from the previous image data 94 stored at the server 14 or theclient device 12 of FIG. 1A. After an aerial image of an item ofinfrastructure is captured, the aerial image may be stored in theprevious image data 94, so that it may be compared with a newly capturedimage of the same infrastructure item at a later date. In this manner,the degradation/maintenance of an infrastructure item may be determinedover time for a more detailed analysis of the damage.

In addition to comparing data points, the comparison module 320 maycompare a set of data points which make up a component of aninfrastructure item to a previously stored image of the same component.For example, a set of data points may display a deck of a bridge, theshoulder on a highway, etc. The entire set may then be compared to datadisplaying, for example, a highway shoulder in good condition todetermine physical differences between the set of data points and thedata displaying the highway shoulder.

After comparisons have been made for the infrastructure item, acondition determination module 330 may determine the conditions ofvarious portions of the infrastructure item. Conditions may includecondition categories such as “poor,” “fair,” “moderate,” “good,”“excellent,” etc., or may include numerical condition scores, forexample from a scale of one to one hundred. For example, a portion of abridge having dents and cracks may be determined to be in poorcondition.

In addition to determining conditions of various portions of theinfrastructure item, the condition determination module 330 maydetermine the size of a portion of the infrastructure item whichrequires repair. For example, the condition determination module 330 maydetermine a portion of the bridge is in “poor” condition, because it isdented. Moreover, the condition determination module 330 may determinethe size of the dent based on the GPS coordinates of an aerial imagedepicting the bridge. This information may be used to determine the costof repairing the dent.

Each data point along with the respective determined condition may thenbe provided to the infrastructure damage severity determination module119 as depicted in FIG. 4 . The infrastructure damage severitydetermination module 119 may include a damage severity leveldetermination module 410, and a damage severity level indicator module420. The damage severity level determination module 410 may determinethat a data point or a set of data points belongs to a particular damageseverity level category from a set of damage severity level categories.For example, the set of damage severity level categories may include“light damage,” “moderate damage,” “severe damage,” “total loss,” “noloss,” “very light damage,” etc. In other embodiments, the damageseverity determination module 410 may determine a score for each datapoint, such as a numeric percentage representing the amount of damage tothe portion of the infrastructure item represented by the data point.For example, the damage severity determination level module 410 maydetermine 100 percent damage to a bridge deck which has split in halfand thus needs to be completely replaced. On the other hand, the damageseverity determination level module 410 may determine 50 percent damageto a large crack or pothole in the road, which can be repaired and doesnot need to be replaced. In some embodiments, the damage severity leveldetermination module 410 may determine a damage severity level categorybased on the numeric percentage representing the amount of damage to theportion of the infrastructure item represented by the data point. Forexample, the damage severity level determination module 410 maydetermine portions of infrastructure items having less than 20 percentdamage or some other predetermined damage threshold are within the“light” category, portions of infrastructure items having between 20 and50 percent damage are within the “moderate” category and portions ofinfrastructure items having more than 50 percent damage are within the“severe” category.

The damage severity level determination module 410 may determine theamount of damage based on the condition of the portion of theinfrastructure item. For example, there may be a higher amount of damagedetermined for a bridge in poor condition than a bridge in excellentcondition. The amount of damage may also be determined based on whethera portion of the infrastructure item needs to be replaced or can berepaired. Portions of infrastructure items requiring replacement maycorrespond to a higher amount of damage than portions of infrastructureitems requiring repair. The damage severity level determination module410 may include a set of rules for determining whether a particularportion of an infrastructure item needs to be repaired or can bereplaced based on its condition. For example, suspension cables on abridge in poor condition may need to be replaced whereas a portion of aroad in the same poor condition may be repaired. The damage severitylevel may then be determined based on the amount of damage, as describedabove.

Once the damage severity level is determined for each data point or eachset of data points, the damage severity level indicator module 420 mayassign a damage severity level indicator to the data point or a set ofdata points. For example, each damage severity level category from theset of damage severity level categories may be assigned a respectivedamage severity level indicator. In some embodiments, the damageseverity level indicator may be a color selected from a set of colors.More specifically, the “moderate” damage severity level category maycorrespond to the color yellow, for example. Moreover, the “severe”damage severity level category may correspond to the color red, and the“light” damage severity level category may correspond to the colorgreen. In other embodiments, a range of damage severity levelpercentages may be assigned a damage severity level indicator. Forexample, damage severity level percentages less than 20 percent maycorrespond to the color green. The corresponding damage severity levelindicators may then be assigned to each data point based on thedetermined damage severity level for the data point. For example, a setof data points representing a portion of a road with moderate damage maybe assigned the color yellow. An assigned damage severity levelindicator for a set of data points may then be appended to one or moreaerial images which may be 3D aerial images and which display thecorresponding data points. For example, an aerial image displaying theportion of the road may display the color yellow overlaying the portionof the road.

While the damage severity level indicators are described as the colorsred, green and yellow, the indicators are not limited to thoseparticular colors. Instead, the damage severity level indicators mayinclude any color and also may include any other suitable representationof a damage severity level. For example, damage severity levelindicators may include numbers which are placed over each data point orset of data points, labels, symbols, different shading techniques, etc.

The aerial images which display infrastructure items and include damageseverity level indicators may then be displayed on the client device 12for an inspector to view. In some embodiments, the client device 12 maydisplay a 3D aerial image of an infrastructure item with damage severitylevel indicators overlaying the image. Moreover, in some embodiments,the client device 12 may display several aerial images for ainfrastructure item and include the damage severity level indicators ineach aerial image.

In some embodiments, the infrastructure evaluation module 72 may includea set of rules for determining a cost estimate based on the damageseverity levels of the various portions or components of aninfrastructure item. For example, the infrastructure evaluation module72 may determine a cost estimate for repairing or replacing portions ofinfrastructure items based on corresponding damage severity levels. Insome embodiments, the set of rules may include a table with apredetermined cost estimate for the different types of infrastructureitems as well as their respective quality (e.g., bridges may be moreexpensive to repair than highways), size (e.g., based on square footage)and damage severity level. For example, the set of rules may include acost estimate of $50,000 for a small road with moderate damage. The setof rules may be stored in the financial database 96. Cost estimates foreach portion of an infrastructure item may be aggregated and/or combinedto determine an overall cost estimate for repairing the damage to theinfrastructure item. In other embodiments, an inspector or a user of theclient device 12 may view the damage severity levels of the variousportions or components of an infrastructure item and determine theappropriate cost estimate for repair. In addition to cost estimates, thedamage severity levels may also be used to determine whether furtherinspection may be necessary, whether a road, highway, bridge, etc., maybe closed for an extended period of time for construction/repairs, etc.

FIG. 5 illustrates an exemplary display 500 of an aerial image of a roadincluding damage severity level indicators. In some embodiments, thedisplay 500 may be presented on the image user interface 70 of theclient device 12 of FIG. 1A. In other embodiments, the display may bepresented on another computing device. The display 500 may include alegend 502 which explains the meaning of each damage severity levelindicator present on the display 500. For example, the legend 502explains a red color indicates “severe” damage (50 percent damage), ayellow color indicates “moderate” damage (30 percent damage), and agreen color indicates “light” damage (20 percent damage). In someembodiments, the green color may indicate less than 20 percent loss, theyellow color may indicate between 20 and 50 percent loss and the redcolor may indicate more than 50 percent loss.

Additionally, the display 500 may include a road 510 as well as terrainsurrounding the road, such as grass, mountains, the sky, clouds, etc.Data points depicting the terrain surrounding the roads may be filteredout by the filtering module 310 of FIG. 3 during the comparison stage.On the other hand, the data points depicting the road 510 may becompared to a previously stored image of the road 510 or a road in goodcondition to determine conditions of portions of the road 510 andresulting damage severity levels. Based on this comparison, a firstportion of the road 504 near the center has a green damage severitylevel indicator, indicating that the center of the road has experienced“light” damage. Surrounding the center of the road, a red damageseverity level indicator overlays a second portion 506, indicating thatthe second portion surrounding the center has experienced “severe”damage. This may be because of large dents or potholes in the pavement.Thus, the second portion 506 may require extensive repairs, whereas thefirst portion may not require any repairs and instead may requirefurther inspection at a later date. However, in some embodiments, theamount of repairs may also depend on the item of infrastructure. Forexample, a tunnel may require replacement when experiencing “severe”damage, while a road may require repair when experiencing the sameamount of loss. Moreover, surrounding the second portion 506 is a thirdportion 508 of the road 510 which does not include a damage severitylevel indicator. The third portion 508 may not have experienced anydamage and therefore may not require any repairs or further inspection.

The aerial image on the display 500 may be just one of several images ofthe road 510. Moreover, while the display 500 depicts a small section ofthe road 510 which ends at a bend, other aerial images may displayadditional sections taken at different angles, locations, and/or zoomlevels than the display 500. The other aerial images may be combinedand/or aggregated to display a larger section of the road 510 depictingseveral bends.

A user such as an inspector or the infrastructure evaluation module 72may determine the amount of damage to each portion of the infrastructureitem to calculate a cost estimate and combine the cost estimates for theportions to determine an overall cost estimate for repairing/replacingthe infrastructure item. Moreover, the user may also determine theamount of damage to each portion to determine whether a road, highway,bridge, etc., may be closed for an extended period of time forconstruction/repairs, etc.

FIG. 6 illustrates a flow diagram representing an exemplary method 600for performing an automatic damage assessment of an item ofinfrastructure. The method 600 may be executed on the client device 12,the server computer 14 or some combination of the client device 12 andthe server computer 14. For example, at least a portion of the method600 may be performed by the infrastructure evaluation module 72 of FIG.1A which as mentioned above, may be disposed on the client device 12,the server computer 14 or some combination of the client device 12 andthe server computer 14. In an embodiment, the infrastructure evaluationmodule 72 may include computer-executable instructions stored on one ormore non-transitory, tangible, computer-readable storage media ordevices, and the computer-executable instructions of the infrastructureevaluation module 72 may be executed to perform the method 600.

At block 602, an item of infrastructure may be identified for assessingdamage. For example, the item of infrastructure may be a highway, aroad, a bridge, a tunnel, a sewer treatment plant, a water treatmentplant, a reservoir, an aqueduct, an electric power grid, acommunications tower, a sidewalk, a paved walkway, a rail line, awaterway (e.g., locks and dams), a port facility, or a publictransportation system. Then, location boundaries for capturing aerialimages of the item of infrastructure may be determined (block 604). Forexample, the location boundaries may be GPS coordinates whichencapsulate the identified item of infrastructure. The locationboundaries may be determined by looking up the location of theidentified infrastructure item in the location database 68.

At block 606, aerial images which display the infrastructure item may bereceived. The aerial images may be received from the satellite device18, the MAV, or the UAV(s) 40 of FIG. 1A. In some embodiments, usercontrols may be disposed on the client device 12 which allow a user,such as an inspector, to control the UAV(s) 40 remotely and determinewhen and where to capture aerial images. In other embodiments, theUAV(s) 40 may be preprogrammed to capture aerial images between thespecified location boundaries. Moreover, in some embodiments, the set ofaerial images may be aggregated to form a 3D display of theinfrastructure item. For example, the set of aerial images may beaggregated using LIDAR, stereoscopy, or photogrammetry techniques tocreate the 3D image.

At block 608, the infrastructure evaluation module 72 may determine thecondition of the infrastructure item based on the aerial images. In someembodiments, the aerial images which depict the infrastructure item maybe made up of data points and a condition may be determined for eachdata point, or alternatively, for each set of data points. The conditionmay be determined by filtering out data points which do not depict theinfrastructure item and comparing the remaining data points to datadepicting a previous image of the infrastructure item taken while theinfrastructure item was in good condition. Additionally, the conditionmay be determined by comparing the remaining data points to datadepicting a similar infrastructure item in good condition andidentifying differences between the two.

Based on the condition of the infrastructure item, a damage severitylevel may be determined (block 610). For example, a damage severitylevel score or a damage severity level category may be determined foreach of the data points depicting the infrastructure item. The damageseverity level score or category may be determined for a data pointbased on the determined condition of a portion of the infrastructureitem depicted by the data point. A damage severity level indicator maythen be assigned to the infrastructure item based on the determineddamage severity level (block 612). For example, data points depicting aportion of an infrastructure item having “light damage” may be assigneda green color indicator. Moreover, in some embodiments, the damageseverity level indicators may overlay the aerial images of theinfrastructure item on a display, for example, on the client device 12(block 614). In addition to placing the damage severity level indicatorsover the aerial images of the infrastructure item, a cost estimate forrepairing and/or replacing the infrastructure item may be determined.For example, the damage severity level for a particular portion of theinfrastructure item may be compared with a set of rules which includescost estimates based on the type of infrastructure item, the quality ofthe infrastructure item, the size of the portion of the infrastructureitem, and/or the damage severity level. Cost estimates for portions ofthe infrastructure item may be aggregated and/or combined to determine atotal cost estimate for repairing the damage to the infrastructure item.

Furthermore, as described above, in some embodiments, items ofinfrastructure may require further inspection, for example, when thedamage severity level is “light damage.” When further inspection isrequired (block 616), the method steps 604 to 614 may be repeated, andthe infrastructure evaluation module 72 may direct the UAV(s) 40, MAV orsatellite device 18 to capture additional aerial images until furtherinspection is no longer necessary. In other embodiments, theinfrastructure evaluation module 72 may direct a person such as aninspector to go to the site and manually inspect the infrastructureitem.

To display real-time (or at least near real-time) aerial images ofinsured properties during a catastrophe, a property display system mayidentify a neighborhood affected by a catastrophe and containing a largeconcentration of properties that are insured by the same insuranceprovider. For example, more than 20 percent of the properties in theneighborhood may be insured by the same insurance provider. Then anautomatic inspection of the entire neighborhood may be performed bycapturing real-time aerial images of the properties in the neighborhood.The automatic inspection may be performed by an unmanned aerial vehicle(UAV), or by a swarm of UAVs working together, which may be controlledby an insurance agent or by the system and flown all over theneighborhood to capture the images. Alternatively, the automaticinspection may be performed by a satellite which also captures real-timeaerial images of the properties within the neighborhood. Moreover, theinspection may also be performed by a manned aerial vehicle (MAV) whichcaptures aerial images of the properties while flying over theneighborhood. Each captured aerial image may be associated with alocation, for example a GPS location, and the GPS location may be usedto determine the owner of the property which is displayed in thecaptured aerial image.

The real-time aerial images for a particular insured property may thenbe transmitted to the property owner or to a user who is approved by theproperty owner, in the form of a web page. For example, the propertyowner may have a customer account with the insurance provider that isaccessible using login information (e.g., a username and password). Theproperty owner may designate authorized users, such as relatives of theproperty owner who can also view the particular insured property throughusing their own login information. In some embodiments, when theproperty is public property such as a public road or public school, allresidents of the city may be designated as authorized users. Anotification may be transmitted to the property owner and/or to theauthorized users, using the property owner's and/or the authorizedusers' contact information (e.g., via email, a short message service(SMS) text message, an alert, a voice recording, etc.), to indicate tothe property owner and/or to the authorized users that a catastrophe hasoccurred in the property owner's neighborhood and that real-time aerialimages of the property have been generated and are accessible throughthe customer accounts. When the property owner and/or the authorizedusers log in to their customer accounts via a web-enabled device such acomputer, laptop, mobile phone, etc., a display of the real-time aerialimages of the property may be provided. The display may include severalimages at different angles, and locations within the neighborhood. Inthis manner, the property owner receives a complete view of thecondition of her property during a catastrophe, even when she is awayfrom her home. Moreover, relatives or close friends of the propertyowner may also receive a complete view of the property, which may allowthem to come to the aid of the property owner more quickly.

Additionally, during the catastrophe the automatic inspection may beperformed several times at predetermined time intervals to provideproperty owners with constant updates of the statuses of theirproperties. For example, the UAV(s), MAV, or satellite may repeatedlysurvey the neighborhood every 30 minutes by capturing the samephotographs and/or video of the neighborhood as captured previously. Theupdated images may then be transmitted to the property owner'sweb-enabled device, thereby providing owners with up-to-the-minuteupdates on the statuses of their properties.

Generally speaking, to display real-time (or at least near real-time)aerial images of insured properties during a catastrophe, an aerialimage capturing device which may be a satellite, MAV or one or severalUAV(s) is/are directed to capture images within an identifiedneighborhood affected by a catastrophe and having a large percentage ofproperties which are insured by an insurance provider. In someembodiments, the aerial image capturing device may also capture imagesfor a neighborhood after a theft or other property damage. The aerialimage capturing device may be directed by a remote control client devicehaving user controls for determining the location and the amount ofphotographs or video captured. The captured aerial images may then beprovided to the remote control client device or to a server computer andgrouped based on their GPS locations to determine a group of aerialimages which correspond to an insured property. Each group of aerialimages corresponding to an insured property may be aggregated, forexample using photogrammetry, stereoscopy, or LIDAR, to create a3-dimensional (3D) image of the insured property.

The 2D or 3D image may be created at a predefined level of detail (e.g.,accurate to within ten percent) and/or may be adjustable (e.g., a useror the system may be able to “zoom in” or “zoom out” of the image).Moreover, the 2D or 3D image may be divided into property components,such as a roof, a door, siding, an exterior wall, a front lawn, abackyard, an outdoor swimming pool, a fence, a tree, a deck, a patio,etc.

The server computer may then transmit a notification to the owner of theinsured property, using the owner's contact information, alerting theowner that a catastrophe, theft or other damage has occurred in herneighborhood and that images of her property have been generated and areavailable for viewing. The notification may be an electronicnotification such as an email, SMS text message or alert, and may alsoinclude a prompt or link to a login page for accessing the images. Insome embodiments, authorized users may also receive the notification.When the owner or another authorized user successfully logs in to thesystem, the server computer may transmit a property display web pagewhich includes real-time (or at least near real-time) aerial images ofthe owner's property. The web page may include several images taken fromdifferent locations, angles and/or zoom levels and may include usercontrols which enable the owner or authorized user to toggle betweeneach of the real-time aerial images of her property. Additionally, asthe owner or authorized user views the real-time aerial images, theaerial image capturing device may capture new aerial images taken fromthe same locations, angles and zoom levels after a predetermined timeinterval. The new aerial images may also be transmitted to the owner orauthorized user, and when the owner or authorized user refreshes thepage or toggles to another image, the most recent aerial images may bedisplayed. In this manner, the owner or authorized user may be providedwith up-to-the-minute updates on the status of the property.

For example, the Smith family may be away from their home in Floridawhen a hurricane strikes their neighborhood. Due to the severity of thestorm, many of the Smiths' neighbors may be left without cell phoneservice and the Smiths may have no way of reaching their neighbors tofind out about the condition of their home. However, the Smith familymay be notified of the hurricane by the property display system viaemail, or an alert on their cell phones. One of the Smiths may then signon to his customer account through his insurance provider and view webpages or application screens which display the status of the Smiths'home. The Smiths may also see periodic updates to the images which maycalm their fears or at least allow them to avoid unexpected surprises.Moreover, viewing images of their home may allow the Smiths to makehotel or other temporary arrangements before they comes home. The imagesmay also allow the Smith family to call repairmen and begin fixing thedamage done to their house as soon as possible.

In an alternative embodiment, the aerial image capturing device may alsocapture aerial images for surveillance. For example, after receivingpermission from a property owner, the aerial image capturing device maycapture aerial images of a car or boat dealership, a department store, arestaurant, a shopping mall, a warehouse, an office building, etc. Theaerial image capturing device may hover over the property and captureaerial images at predetermined time intervals (e.g. every minute, everysecond, etc.). Moreover, a notification may be transmitted to the ownerwhen suspicious activity has occurred prompting the owner to log in tothe system. When the owner successfully logs in, the server computer maytransmit a property display web page which includes real-time (or atleast near real-time) aerial images of the dealership, department store,restaurant, office building, etc. In this manner, additionalsurveillance may be provided from the exterior of a building. Moreoverin some embodiments, the captured aerial images may be transmitted to asecurity company which may analyze the aerial images in addition toimages from their own security cameras.

The user may launch a client application from the web-enabled device viaany suitable manner, such as touch-selecting a client application iconon the display of the web-enabled device, double-clicking on the clientapplication icon via a mouse of the web-enabled device or a trackpad ofthe web-enabled device. After the user launches the client application,the home screen of the client application is displayed to the user onthe web-enabled device.

The home screen may include a notification alerting the user that adisaster has impacted her neighborhood or the neighborhood of a propertyowner who authorized the user to view the property. For example, thenotification may state, “Alert: A disaster has impacted yourneighborhood! Login to view images of your property.” The home screenmay also include an “OK” button which when selected by the user, maydirect the user to a login screen. In some embodiments, the notificationmay be embedded over the login screen, and when the “OK” button isselected, the notification may disappear. In such an embodiment, a“Login” button may appear on the home screen.

In order to receive the notification at the client application beforethe user enters login information, the data storage of the web-enableddevice may store application data for the client application, asdescribed above. For example, this data may include the location of theuser's property, a username or any other suitable information from whichto identify the user of the web-enabled device. While the login screenmay require an enhanced level of security, for example, requiring theuser to enter both a username and a password, the notification may betransmitted upon verifying the identity of the user based on the storedapplication data. For example, the server may determine based on acomparison of the GPS location of the aerial images to location data inthe location database, that an aerial image displays John Doe's home.The server may then receive application data from a web-enabled devicewhich includes John Doe's username, and as a result, the server maytransmit the notification to the web-enabled device. In otherembodiments, the user remains logged in after initially logging in tothe client application and the notification may be transmitted byverifying the user's login information.

Moreover, in some embodiments where the web-enabled device is a mobilesmart phone, the notification may appear on the lock screen of themobile smart phone even before the user selects the client application.Additionally, the notification may also appear again on the home screenwhen the user selects the client application.

Further, in yet other embodiments, the notification may be transmittedto the user via email, SMS message, an automated voice recording, etc.The server may transmit the notification by identifying contactinformation for the owner of the insured property. For example, contactinformation such as a phone number or email address may be stored in thecustomer database as part of a customer account for the owner. Theserver may then transmit the notification to a device associated withthe identified phone number, or to the identified email address. In someembodiments, the owner may enter the contact information into hercustomer account, or contact information may be obtained when the ownertakes out an insurance policy with the insurance provider. In suchinstances, the notification may appear again on the home screen when theuser selects the client application or may not appear on the clientapplication.

A login screen may include an area for logging in to a customer accountby entering a username and a password. The login screen may also includean area for remembering the username to avoid entering the same usernamethe next time the user logs in to the system. Once the user enters ausername and password, the user may select a “Login” button. After the“Login” button is selected, the server may verify that the username andpassword correspond to a customer profile from the customer database. Ifthe username and password do not correspond to a customer account, thelogin screen may display an error message and prompt the user to reentera username and password. If there is a successful match, the clientapplication may display a property display screen.

The property display screen may include the name of the property beingdisplayed, i.e., “Your Home,” the name of the aerial image beingdisplayed or the viewpoint of the property, i.e., “Front View,” and anaerial image which displays the property. The property display screenmay also include “Next” and “Back” buttons which when selected allow theuser to toggle between each of the aerial images which display theproperty. For example, the user may be John Doe and the property may beJohn Doe's home. In another example, when the user is authorized by theproperty owner to view the property owner's home, the name of theproperty being displayed may be, “John Doe's Home” rather than “YourHome.” In some embodiments, in addition to displaying homes, theproperty display screen may display a business or office building ownedby the user. In any event, when John Doe selects the “Next” button, anaerial image which displays a rear view of his property may be displayedand the name of the aerial image may be “Rear View.” Moreover, when Johnselects the “Back” button, an aerial image which displays an overheadview of his property may be displayed and appropriately named “OverheadView.” In some embodiments, the property display screen may includeaerial images of the exterior of a commercial building for surveillance.

When the UAV 40, the MAV, or the satellite device 18 captures a newaerial image of the same property and at the same viewpoint, theproperty display screen may update the property display with the newaerial image. For example, if the satellite device 18 takes a new frontview image of John Doe's house, the display of the “Front View” may beremoved and replaced with the new aerial image.

If a user has multiple properties in a neighborhood affected by acatastrophe or in multiple neighborhoods each affected by a catastrophe,the user may toggle between the different properties. For example, theuser may select the triangle button to select another property than“Your Home,” such as “Your Business.” When the user selects a differentproperty name, several aerial images which display the differentproperty may appear on the property display along with their respectivenames, such as a front view of John Doe's business, a rear view of JohnDoe's business, a side view of John Doe's business, etc. The propertydisplay may include an apartment building, a condominium, a mobile home,a house boat, a vacation home, a boat, an RV, a hotel, commercialbuildings such as an office building or business, a store or arestaurant or any other real estate property which may be insured by aninsurance provider and may include residences or docks/storage.

In some embodiments, the property display may further include indicatorsoverlaying the property which indicate the extent of the damage to theproperty (also referred to interchangeably herein as “damage severitylevel indicators”). The property may be divided into property componentssuch as a roof, a door, a window, siding, exterior walls, a lawn, abackyard, a driveway, a garage, an outdoor swimming pool, a fence, atree, a deck, a patio, etc. A different damage severity level indicatormay be assigned to each property component depending on the extent ofthe damage to the property component. For example, the damage severitylevel indicators may be a set of colors where each color represents adifferent damage severity level. The property display of John Doe's homemay include a red roof, a green front door, yellow windows and purplesiding. The color indicators may further help the user understand theextent of the damage to her property.

While the damage severity level indicators are described as the colorsred, green, purple, and yellow, the indicators are not limited to thoseparticular colors. Instead, the damage severity level indicators mayinclude any color and also may include any other suitable representationof a damage severity level. For example, damage severity levelindicators may include numbers which are placed over each propertycomponent, labels, symbols, different shading techniques, etc.

The aerial images may be received from the UAV(s) 40 by automaticallydirecting the one or several UAV(s) 40 to fly within the set ofboundaries which encapsulate the identified neighborhood. The UAV(s) 40may also be directed to take several photographs or capture video atdifferent locations within the neighborhood and at several angles.Moreover, the UAV(s) 40, the MAV, or the satellite device 18 may bedirected to repetitively capture the same images after a predeterminedtime interval. For example, the UAV(s) 40, the MAV, or satellite device18 may be directed to capture the same images every 5 minutes, every 10minutes, every hour, every 3 hours, etc. Alternatively, after theneighborhood is identified, a remote control client 12 user such as aninsurance agent may control the UAV(s) 40 remotely, through a series ofuser controls on the user interface 30 to cause the UAV(s) 40 to takepictures and/or video at different locations within the neighborhood andat several angles. In other embodiments, the property display generator64 may request and/or receive aerial images of the exterior of acommercial property, which may be received from the satellite device 18or the UAV(s) 40. The UAV(s) 40 may be automatically directed to hoverover the a set of boundaries which encapsulates the commercial property.Moreover, the UAV(s) 40, the MAV or the satellite device 18 may bedirected to repetitively capture the same images after a predeterminedtime interval (e.g., every second, every minute, every hour, etc.).

After the aerial images are captured and received for the identifiedneighborhood, the property display generator may filter out aerialimages that do not display insured properties, and may group togetherall of the aerial images which display a single insured property. Forexample, the customer data and the location data stored in the customerdatabase and the location database of the server, respectively, may beused to determine the locations of insured properties as well as theirrespective property owners. The locations of insured properties may becompared to a received aerial image which contains GPS coordinates ofits data points, as described above, to determine whether the receivedaerial image displays an insured property. For example, if the locationof an aerial image matches with one of the locations of the insuredproperties then the aerial image displays an insured property. If thereceived aerial image does not display any insured properties the aerialimage may be discarded. In some embodiments, none of the aerial imagesdisplaying the neighborhood are discarded, and all of the aerial imagesare utilized. In any event, the property display generator may group theremaining received aerial images with other aerial images which displaythe same property. In some embodiments, an aerial image may display morethan one property. In this instance, the aerial image may be groupedwith each of the properties that the image displays.

Each group of aerial images which displays the same property may becombined. The group of aerial images may be combined to generate a 3Dimage of the property using 3D imaging techniques such as stereoscopy,LIDAR, or photogrammetry. The property display generator may utilize theCartesian or GPS coordinates received with each aerial image toreconstruct a 3D image of the property using the group of aerial imagescaptured at different locations and angles. Each group of aerial imagesmay be combined to generate a 3D aerial image of each property includingeach insured property in the neighborhood. The 3D aerial image may becreated at a predefined level of detail (e.g., accurate to within tenpercent) and/or may be adjustable (e.g., a user or the system may beable to “zoom in” or “zoom out”).

The property display generator may then identify the owner of theproperty displayed in the 3D aerial image or the group of aerial imagesby accessing the customer database and the location database.Additionally, the property display generator may also generate a webpage displaying the 3D aerial image or group of aerial images, which maybe transmitted to the owner upon receiving login information for acustomer account.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Additionally, certain embodiments are described herein as includinglogic or a number of routines, subroutines, applications, orinstructions. These may constitute either software (e.g., code embodiedon a machine-readable medium or in a transmission signal) or hardware.In hardware, the routines, etc., are tangible units capable ofperforming certain operations and may be configured or arranged in acertain manner. In example embodiments, one or more computer systems(e.g., a standalone, client or server computer system) or one or morehardware modules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connect the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods or routines described herein may be at leastpartially processor-implemented. For example, at least some of theoperations of a method may be performed by one or more processors orprocessor-implemented hardware modules. The performance of certain ofthe operations may be distributed among the one or more processors, notonly residing within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the description. Thisdescription, and the claims that follow, should be read to include oneor at least one and the singular also includes the plural unless it isobvious that it is meant otherwise.

This detailed description is to be construed as exemplary only and doesnot describe every possible embodiment, as describing every possibleembodiment would be impractical, if not impossible. One could implementnumerous alternate embodiments, using either current technology ortechnology developed after the filing date of this application.

We claim:
 1. A method for surveying a property using an unmanned aerialvehicle (UAV), the method comprising: identifying, by one or moreprocessors on a computing device, a commercial property for the UAV toperform surveillance; directing, by the one or more processors, the UAVto hover over the commercial property and capture, via an imagingdevice, aerial images at predetermined time intervals; receiving, by theone or more processors, the aerial images of the commercial propertycaptured at the predetermined time intervals; determining, by the one ormore processors, a damage severity level for the commercial propertybased on the aerial images; generating, by the one or more processors, asurveillance alert; transmitting, by the one or more processors, thesurveillance alert to an electronic device associated with an authorizeduser associated with the commercial property; and upon receiving andauthenticating user login information to confirm that a user of theelectronic device is the authorized user associated with the commercialproperty, transmitting, by the one or more processors, a propertydisplay web page to the electronic device, the property display web pageincluding a damage severity level indicator displayed overlaying theaerial images of the commercial property.
 2. The method of claim 1,further comprising directing, by the one or more processors, the UAV toone or more locations for capturing the aerial images.
 3. The method ofclaim 1, further comprising filtering, by the one or more processors,the aerial images to remove portions which do not depict the commercialproperty.
 4. The method of claim 1, further comprising: aggregating, bythe one or more processors, the aerial images to generate a threedimensional display of the commercial property; and transmitting, by theone or more processors, the three dimensional display of the commercialproperty to the electronic device associated with the owner of thecommercial property.
 5. The method of claim 1, wherein the propertydisplay web page further includes real-time aerial images of thecommercial property.
 6. The method of claim 5, further comprising:determining, by the one or more processors, a predetermined period oftime for updating the property display page; and when the predeterminedperiod of time has elapsed: receiving, at the one or more processors,one or more new real-time aerial images of the commercial property;updating, by the one or more processors, the property display page withthe new real-time aerial images; and transmitting, by the one or moreprocessors, the updated property display page to the electronic device.7. The method of claim 1, wherein the surveillance alert includes atleast one of: an email, a short message service (SMS) message, or anautomated voice recording.
 8. A system for surveying a property using anunmanned aerial vehicle (UAV), the system comprising: a communicationnetwork; and one or more computing devices communicatively coupled tothe communication network, each of the one or more computing deviceshaving a memory and one or more processors and at least one of thecomputing devices configured to: identify a commercial property for theUAV to perform surveillance; direct the UAV to hover over the commercialproperty and capture, via an imaging device, aerial images atpredetermined time intervals; receive, via the communication network,the aerial images of the commercial property captured at thepredetermined time intervals; determine a damage severity level for thecommercial property based on the aerial images; generate a surveillancealert; transmit, over the communication network, the surveillance alertto an electronic device associated with an authorized user associatedwith the commercial property; upon receiving and authenticating userlogin information to confirm that a user of the electronic device is theauthorized user associated with the commercial property, transmit, overthe communication network, a property display web page to the electronicdevice, the property display web page including a damage severity levelindicator displayed overlaying real-time aerial images of the commercialproperty.
 9. The system of claim 8, wherein the at least one computingdevice is further configured to direct the UAV to one or more locationsfor capturing the aerial images.
 10. The system of claim 8, wherein theat least one computing device is further configured to filter the aerialimages to remove portions which do not depict the commercial property.11. The system of claim 8, wherein the at least one computing device isfurther configured to: aggregate the aerial images to generate a threedimensional display of the commercial property; and transmit, over thecommunication network, the three dimensional display of the commercialproperty to the electronic device associated with the owner of thecommercial property.
 12. The system of claim 8, wherein the at least onecomputing device is further configured to: determine a predeterminedperiod of time for updating the property display page; and when thepredetermined period of time has elapsed: receive one or more newreal-time aerial images of the commercial property; update the propertydisplay page with the new real-time aerial images; and transmit, overthe communication network, the updated property display page to theelectronic device.
 13. The system of claim 8, wherein the surveillancealert includes at least one of: an email, a short message service (SMS)message, or an automated voice recording.
 14. A non-transitorycomputer-readable memory storing executable instructions that, whenexecuted by one or more processors, cause the one or more processors to:identify a commercial property for a UAV to perform surveillance; directthe UAV to hover over the commercial property and capture, via animaging device, aerial images at predetermined time intervals; receive,via the communication network, the aerial images of the commercialproperty captured at the predetermined time intervals; determine adamage severity level for the commercial property based on the aerialimages; generate a surveillance alert; transmit the surveillance alertto an electronic device associated with an authorized user associatedwith the commercial property; and upon receiving and authenticating userlogin information to confirm that a user of the electronic device is theauthorized user associated with the commercial property, transmit aproperty display web page to the electronic device, the property displayweb page including a damage severity level indicator displayedoverlaying the aerial images of the commercial property.
 15. Thenon-transitory computer-readable memory of claim 14, wherein theinstructions further cause the one or more processors to direct the UAVto one or more locations for capturing the aerial images.
 16. Thenon-transitory computer-readable memory of claim 14, wherein theinstructions further cause the one or more processors to filter theaerial images to remove portions which do not depict the commercialproperty.
 17. The non-transitory computer-readable memory of claim 14,wherein the instructions further cause the one or more processors to:aggregate the aerial images to generate a three dimensional display ofthe commercial property; and transmit, over the communication network,the three dimensional display of the commercial property to theelectronic device associated with the owner of the commercial property.18. The non-transitory computer-readable memory of claim 14, wherein theproperty display web page further includes real-time aerial images ofthe commercial property.
 19. The non-transitory computer-readable memoryof claim 18, wherein the instructions further cause the one or moreprocessors to: determine a predetermined period of time for updating theproperty display page; and when the predetermined period of time haselapsed: receive one or more new real-time aerial images of thecommercial property; update the property display page with the newreal-time aerial images; and transmit, over the communication network,the updated property display page to the electronic device.